An Experimental Set of Hand Gestures for Expressive Control of Musical Parameters in Realtime

This paper describes the implementation of Time Delay Neural Networks (TDNN) to recognize gestures from video images. Video sources are used because they are non-invasive and do not inhibit performer's physical movement or require specialist devices to be attached to the performer which experience has shown to be a significant problem that impacts musicians performance and can focus musical rehearsals and performances upon technical rather than musical concerns (Myatt 2003). We describe a set of hand gestures learned by an artificial neural network to control musical parameters expressively in real time. The set is made up of different types of gestures in order to investigate: • aspects of the recognition process • expressive musical control • schemes of parameter mapping • generalization issues for an extended set for musical control The learning procedure of the Neural Network is described which is based on variations by affine transformations of image sequences of the hand gestures. The whole application including the gesture capturing is implemented in jMax to achieve real time conditions and easy integration into a musical environment to realize different mappings and routings of the control stream. The system represents a practice-based research using actual music models like compositions and processes of composition which will follow the work described in the paper.